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Abstract 

We  propose  new  methods  to  take  advan¬ 
tage  of  text  in  resource-rich  languages 
to  sharpen  statistical  language  models  in 
resource-deficient  languages.  We  achieve 
this  through  an  extension  of  the  method 
of  lexical  triggers  to  the  cross-language 
problem,  and  by  developing  a  likelihood- 
based  adaptation  scheme  for  combining 
a  trigger  model  with  an  JV-gram  model. 

We  describe  the  application  of  such  lan¬ 
guage  models  for  automatic  speech  recog¬ 
nition.  By  exploiting  a  side-coipus  of  con¬ 
temporaneous  English  news  articles  for 
adapting  a  static  Chinese  language  model 
to  transcribe  Mandarin  news  stories,  we 
demonstrate  significant  reductions  in  both 
peiplexity  and  recognition  errors.  We 
also  compare  our  cross-lingual  adaptation 
scheme  to  monolingual  language  model 
adaptation,  and  to  an  alternate  method  for 
exploiting  cross-lingual  cues,  via  cross- 
lingual  information  retrieval  and  machine 
translation,  proposed  elsewhere. 

1  Data  Sparseness  in  Language  Modeling 

Statistical  techniques  have  been  remarkably  suc¬ 
cessful  in  automatic  speech  recognition  (ASR)  and 
natural  language  processing  (NLP)  over  the  last  two 
decades.  This  success,  however,  depends  crucially 
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on  the  availability  of  accurate  and  large  amounts 
of  suitably  annotated  training  data  and  it  is  difficult 
to  build  a  usable  statistical  model  in  their  absence. 
Most  of  the  success,  therefore,  has  been  witnessed 
in  the  so  called  resource-rich  languages.  More  re¬ 
cently,  there  has  been  an  increasing  interest  in  lan¬ 
guages  such  as  Mandarin  and  Arabic  for  ASR  and 
NLP,  and  data  resources  arc  being  created  for  them 
at  considerable  cost.  The  data-resource  bottleneck, 
however,  is  likely  to  remain  for  a  majority  of  the 
world’s  languages  in  the  foreseeable  future. 

Methods  have  been  proposed  to  bootstrap  acous¬ 
tic  models  for  ASR  in  resource  deficient  languages 
by  reusing  acoustic  models  from  resource-rich  lan¬ 
guages  (Schultz  and  Waibel,  1998;  Byrne  et  al., 
2000).  Morphological  analyzers,  noun-phrase  chun¬ 
kers,  POS  taggers,  etc.,  have  also  been  developed 
for  resource  deficient  languages  by  exploiting  trans¬ 
lated  or  parallel  text  (Yarowsky  et  al.,  2001).  Khu¬ 
danpur  and  Kim  (2002)  recently  proposed  using 
cross-lingual  information  retrieval  (CLIR)  and  ma¬ 
chine  translation  (MT)  to  improve  a  statistical  lan¬ 
guage  model  (LM)  in  a  resource-deficient  language 
by  exploiting  copious  amounts  of  text  available  in 
resource-rich  languages.  When  transcribing  a  news 
story  in  a  resource-deficient  language,  their  core 
idea  is  to  use  the  first  pass  output  of  a  rudimentary 
ASR  system  as  a  query  for  CLIR,  identify  a  contem¬ 
poraneous  English  document  on  that  news  topic,  fol¬ 
lowed  by  MT  to  provide  a  rough  translation  which, 
even  if  not  fluent,  is  adequate  to  update  estimates  of 
word  frequencies  and  the  LM  vocabulary.  They  re¬ 
port  up  to  a  28%  reduction  in  peiplexity  on  Chinese 
text  from  the  Hong  Kong  News  coipus. 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

2QQ3  2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2003  to  00-00-2003 

4.  TITLE  AND  SUBTITLE 

Cross-Lingual  Lexical  Triggers  in  Statistical  Language  Modeling 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

John  Hopkins  University, Center  for  Language  and  Speech 

Processing, Department  of  Computer  Science, Baltimore, MD, 21218 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

18.  NUMBER  19a.  NAME  OF 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE 

unclassified  unclassified  unclassified 

8 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


In  spite  of  their  considerable  success,  some  short¬ 
comings  remain  in  the  method  used  by  Khudanpur 
and  Kim  (2002).  Specifically,  stochastic  translation 
lexicons  estimated  using  the  IBM  method  (Brown 
et  al.,  1993)  from  a  fairly  large  sentence-aligned 
Chinese-English  parallel  coipus  arc  used  in  their  ap¬ 
proach  —  a  considerable  demand  for  a  resource- 
deficient  language.  It  is  suggested  that  an  easier- 
to-obtain  document-aligned  comparable  coipus  may 
suffice,  but  no  results  are  reported.  Furthermore,  for 
each  Mandarin  news  story,  the  single  best  match¬ 
ing  English  article  obtained  via  CLIR  is  translated 
and  used  for  priming  the  Chinese  LM,  no  matter 
how  good  the  CLIR  similarity,  nor  are  other  well¬ 
matching  English  articles  considered.  This  issue 
clearly  deserves  further  attention.  Finally,  ASR  re¬ 
sults  are  not  reported  in  their  work,  though  their  pro¬ 
posed  solution  is  clearly  motivated  by  an  ASR  task. 
We  address  these  three  issues  in  this  paper. 

Section  2  begins,  for  the  sake  of  completeness, 
with  a  review  of  the  cross-lingual  story-specific  LM 
proposed  by  Khudanpur  and  Kim  (2002).  A  notion 
of  cross-lingual  lexical  triggers  is  proposed  in  Sec¬ 
tion  3,  which  overcomes  the  need  for  a  sentence- 
aligned  parallel  coipus  for  obtaining  translation  lex¬ 
icons.  After  a  brief  detour  to  describe  topic- 
dependent  LMs  in  Section  4,  a  description  of  the 
ASR  task  is  provided  in  Section  5,  and  ASR  results 
on  Mandarin  Broadcast  News  arc  presented  in  Sec¬ 
tion  6.  The  issue  of  how  many  English  articles  to 
retrieve  and  translate  into  Chinese  is  resolved  by  a 
likelihood-based  scheme  proposed  in  Section  6.1. 

2  Cross-Lingual  Story-Specific  LMs 

For  the  sake  of  illustration,  consider  the  task  of 
sharpening  a  Chinese  language  model  for  transcrib¬ 
ing  Mandarin  news  stories  by  using  a  large  coipus 
of  contemporaneous  English  newswire  text.  Man¬ 
darin  Chinese  is,  of  course,  not  resource -deficient 
for  language  modeling  —  100s  of  millions  of  words 
are  available  on-line.  However,  we  choose  it  for  our 
experiments  partly  because  it  is  sufficiently  different 
from  English  to  pose  a  real  challenge,  and  because 
the  availability  of  large  text  corpora  in  fact  permits 
us  to  simulate  controlled  resource  deficiency. 

Let  df,...,df;  denote  the  text  of  N  test  sto¬ 
ries  to  be  transcribed  by  an  ASR  system,  and  let 


df, . . . ,  df  denote  their  corresponding  or  aligned 
English  newswire  articles.  Correspondence  here 
does  not  imply  that  the  English  document  df  needs 
to  be  an  exact  translation  of  the  Mandarin  story  df . 
It  is  quite  adequate,  for  instance,  if  the  two  stories  re¬ 
port  the  same  news  event.  This  approach  is  expected 
to  be  helpful  even  when  the  English  document  is 
merely  on  the  same  general  topic  as  the  Mandarin 
story,  although  the  closer  the  content  of  a  pair  of  ar¬ 
ticles  the  better  the  proposed  methods  are  likely  to 
work.  Assume  for  the  time  being  that  a  sufficiently 
good  Chinese-English  story  alignment  is  given. 

Assume  further  that  we  have  at  our  disposal  a 
stochastic  translation  dictionary  —  a  probabilistic 
model  of  the  form  Pr(c\e)  —  which  provides  the 
Chinese  translation  c  G  C  of  each  English  word 
e  €  £ ,  where  C  and  £  respectively  denote  our  Chi¬ 
nese  and  English  vocabularies. 

2.1  Computing  a  Cross-Lingual  Unigram  LM 

Let  P(e\df)  denote  the  relative  frequency  of  a  word 
e  in  the  document  df,  eE£,l<i<N.lt  seems 
plausible  that,  V  c  €  C, 

CL — unigram ( C | df  )  =  ]T  PT(c|e)P(e|df  )  ,  (1) 

edS 

would  be  a  good  unigram  model  for  the  «-th  Man¬ 
darin  story  df .  We  use  this  cross-lingual  unigram 
statistic  to  sharpen  a  statistical  Chinese  LM  used  for 
processing  the  test  story  df .  One  way  to  do  this  is 
via  lineal-  interpolation 

-PcL— interpolated  Cfc_2,  dj  )  =  (2) 

APcl  —unigram  (ck\df)  +  (1  -  A)P(c*|c*_i,c*_2) 

of  the  cross-lingual  unigram  model  (1)  with  a  static 
trigram  model  for  Chinese,  where  the  interpolation 
weight  A  may  be  chosen  off-line  to  maximize  the 
likelihood  of  some  held-out  Mandarin  stories.  The 
improvement  in  (2)  is  expected  from  the  fact  that 
unlike  the  static  text  from  which  the  Chinese  trigram 
LM  is  estimated,  df  is  semantically  close  to  df  and 
even  the  adjustment  of  unigram  statistics,  based  on 
a  stochastic  translation  model,  may  help. 

Figure  1  shows  the  data  flow  in  this  cross-lingual 
LM  adaptation  approach,  where  the  output  of  the 
first  pass  of  an  ASR  system  is  used  by  a  CLIR  sys¬ 
tem  to  find  an  English  document  df,  an  MT  system 


Figure  1:  Story-Specific  Cross-Lingual  Adaptation 
of  a  Chinese  Language  Model  using  English  Text. 


computes  the  statistic  of  (1),  and  the  ASR  system 
uses  the  LM  of  (2)  in  a  second  pass. 

2.2  Obtaining  Matching  English  Documents 

To  illustrate  how  one  may  obtain  the  English  doc¬ 
ument  df  to  match  a  Mandarin  story  dp,  let  us 
assume  that  we  also  have  a  stochastic  reverse- 
translation  lexicon  Pr(e|c).  One  obtains  from  the 
first  pass  ASR  output,  cf.  Figure  1,  the  relative  fre¬ 
quency  estimate  P(c\df)  of  Chinese  words  c  in  df, 
c  €  C,  and  uses  the  translation  lexicon  Py(e|c)  to 
compute,  Ve  €  £, 

PCL  —unigram  (e|df)  =  ^PT(e|c)P(c|df),  (3) 
cec 

an  English  bag-of-words  representation  of  the  Man¬ 
darin  story  df  as  used  in  standard  vector-based  in¬ 
formation  retrieval.  The  document  with  the  highest 
TF-IDF  weighted  cosine-similarity  to  df  is  selected: 

dj  —  arg  max  s i in ( TVjl — unigram ( ^ I dj  ),P(e|dj  )) . 

dj 

Readers  familiar  with  information  retrieval  litera¬ 
ture  will  recognize  this  to  be  the  standard  query- 
translation  approach  to  CLIR. 

2.3  Obtaining  Stochastic  Translation  Lexicons 

The  translation  lexicons  Pr(c|e)  and  P/-(ejc)  may 
be  created  out  of  an  available  electronic  translation 
lexicon,  with  multiple  translations  of  a  word  being 


treated  as  equally  likely.  Stemming  and  other  mor¬ 
phological  analyses  may  be  applied  to  increase  the 
vocabulary-coverage  of  the  translation  lexicons. 

Alternately,  they  may  also  be  obtained  auto¬ 
matically  from  a  parallel  coipus  of  translated  and 
sentence-aligned  Chinese-English  text  using  statisti¬ 
cal  machine  translation  techniques,  such  as  the  pub¬ 
licly  available  GIZA++  tools  (Och  and  Ney,  2000), 
as  done  by  Khudanpur  and  Kim  (2002).  Unlike  stan¬ 
dard  MT  systems,  however,  we  apply  the  translation 
models  to  entire  articles,  one  word  at  a  time,  to  get  a 
bag  of  translated  words  —  cf.  (1)  and  (3). 

Finally,  for  truly  resource  deficient  languages,  one 
may  obtain  a  translation  lexicon  via  optical  character 
recognition  from  a  printed  bilingual  dictionary  (cf. 
Doerman  et  al  (2002)).  This  task  is  arguably  easier 
than  obtaining  a  large  LM  training  coipus. 

3  Cross-Lingual  Lexical  Triggers 

It  seems  plausible  that  most  of  the  information  one 
gets  from  the  cross-lingual  unigram  LM  of  (1)  is 
in  the  form  of  the  altered  statistics  of  topic-specific 
Chinese  words  conveyed  by  the  statistics  of  content¬ 
bearing  English  words  in  the  matching  story.  The 
translation  lexicon  used  for  obtaining  the  informa¬ 
tion,  however,  is  an  expensive  resource.  Yet,  if  one 
were  only  interested  in  the  conditional  distribution 
of  Chinese  words  given  some  English  words,  there 
is  no  reason  to  require  translation  as  an  intermedi¬ 
ate  step.  In  a  monolingual  setting,  the  mutual  infor¬ 
mation  between  lexical  pairs  co-occurring  anywhere 
within  a  long  “window”  of  each-other  has  been  used 
to  capture  statistical  dependencies  not  covered  by 
iV-gram  LMs  (Rosenfeld,  1996;  Tillmann  and  Ney, 
1997).  We  use  this  inspiration  to  propose  the  follow¬ 
ing  notion  of  cross-lingual  lexical  triggers. 

In  a  monolingual  setting,  a  pair  of  words  (a,  b )  is 
considered  a  trigger-pair  if,  given  a  word-position  in 
a  sentence,  the  occurrence  of  a  in  any  of  the  pre¬ 
ceding  word-positions  significantly  alters  the  (con¬ 
ditional)  probability  that  the  following  word  in  the 
sentence  is  b:  a  is  said  to  trigger  b.  E.g.  the  occur¬ 
rence  of  either  significantly  increases  the  proba¬ 
bility  of  or  subsequently  in  the  sentence.  The  set  of 
preceding  word-positions  is  variably  defined  to  in¬ 
clude  all  words  from  the  beginning  of  the  sentence, 
paragraph  or  document,  or  is  limited  to  a  fixed  num- 


ber  of  preceding  words,  limited  of  course  by  the  be¬ 
ginning  of  the  sentence,  paragraph  or  document. 

In  the  cross-lingual  setting,  we  consider  a  pair  of 
words  (e,c),  e  E  €  and  c  E  C,  to  be  a  trigger-pair 
if,  given  an  English-Chinese  pair  of  aligned  docu¬ 
ments,  the  occurrence  of  e  in  the  English  document 
significantly  alters  the  (conditional)  probability  that 
the  word  c  appears  in  the  Chinese  document:  e  is 
said  to  trigger  c.  It  is  plausible  that  translation-pairs 
will  be  natural  candidates  for  trigger-pairs.  It  is, 
however,  not  necessary  for  a  trigger-pair  to  also  be  a 
translation-pair.  E.g.,  the  occurrence  of  Belgrade 
in  the  English  document  may  trigger  the  Chinese 
transliterations  of  Serbia  and  Kosovo,  and  pos¬ 
sibly  the  translations  of  China,  embassy  and 
bomb!  By  infering  trigger-pairs  from  a  document- 
aligned  coipus  of  Chinese-English  articles,  we  ex¬ 
pect  to  be  able  to  discover  semantically-  or  topically- 
related  pairs  in  addition  to  translation  equivalences. 


3.1  Identification  of  Cross-Lingual  Triggers 

Average  mutual  information,  which  measures  how 
much  knowing  the  value  of  one  random  variable 
reduces  the  uncertainty  of  about  another,  has  been 
used  to  identify  trigger-pairs.  We  compute  the  av¬ 
erage  mutual  information  for  every  English-Chinese 
word  pair  (e,  c)  as  follows. 

Let  now  be  a  document- 

aligned  training  coipus  of  English-Chinese  article 
pairs.  Let  #d(e,  c)  denote  the  document  frequency, 
i.e.,  the  number  of  aligned  article-pairs,  in  which  e 
occurs  in  the  English  article  and  c  in  the  Chinese. 
Let  #d(e,  c)  denote  the  number  of  aligned  article- 
pairs  in  which  e  occurs  in  the  English  articles  but  c 
does  not  occur  in  the  Chinese  article.  Let 


P(e,c) 


#d(e,  c) 
N 


and  P(e,  c) 


#d(e,c) 

N 


The  quantities  P(e,  c )  and  P(e,  c)  arc  similarly  de¬ 
fined.  Next  let  #d(e)  denote  the  number  of  English 
articles  in  which  e  occurs,  and  define 


+  P(e,  c)  log  +  P(e,  c )  log  . 

We  propose  to  select  word  pairs  with  high  mutual 
information  as  cross-lingual  lexical  triggers. 

There  arc  \£\  x  \C\  possible  English-Chinese  word 
pairs  which  may  be  prohibitively  large  to  search 
for  the  pairs  with  the  highest  mutual  information. 
We  filter  out  infrequent  words  in  each  language, 
say,  words  appealing  less  than  5  times,  then  mea¬ 
sure  I  (e;  c)  for  all  possible  pairs  from  the  remaining 
words,  sort  them  by  I(e;  c),  and  select,  say,  the  top 
1  million  pairs. 


3.2  Estimating  Trigger  LM  Probabilities 

Once  we  have  chosen  a  set  of  trigger-pairs,  the  next 
step  is  to  estimate  a  probability  Prrig(c|e)  in  lieu 
of  the  translation  probability  Pp(c|e)  in  (1),  and  a 
probability  Prrig(e|c)  in  (3). 

Following  the  maximum  likelihood  approach  pro¬ 
posed  by  Tillman  and  Ney  (1997),  one  could  choose 
the  trigger  probability  PTrig(c|e)  to  be  based  on  the 
unigram  frequency  of  c  among  Chinese  word  tokens 
in  that  subset  of  aligned  documents  df  which  have 
e  in  df ,  namely 


Prrig(c|e) 


T,i:df3eNdf:(C) 
T,c'eC  £i  :  df3eNdc(c') 


(4) 


As  an  ad  hoc  alternative  to  (4),  we  also  use 


Prrig(c|e) 


4(e;c) 

£c'ecJ(e;c')  ’ 


(5) 


where  we  set  I(e;c)  =  0  whenever  (e,  c)  is  not  a 
trigger-pair,  and  find  it  to  be  somewhat  more  effec¬ 
tive  (cf.  Section  6.2).  Thus  (5)  is  used  henceforth  in 
this  paper.  Analogous  to  (1),  we  set 

Prrig-unigram(c|df)  =  ^  PTrig(c|e)P(e|df )  ,  (6) 

e£f 


and,  again,  we  build  the  interpolated  model 


P(e) 


#d(e) 

N 


and  P(c|e) 


P(e,c) 

P(e) 


Similarly  define  P(e),  P(c|e)  via  the  document  fre¬ 
quency  #d(e)  =  N  —  #d(e);  define  P(c)  via  the 
document  frequency  #d(c),  etc.  Finally,  let 


I(e;c)  = 


P(e,  c)  log 


P(c\e) 

“PpT 


+  P(e,  c)  log 


-P(cje) 

~PW 


Pltig— interpolated  (Cfc  C/.—2,  dj  )  —  (7) 

APfrig  —unigram  (ck | df )  +  (1  -  A)P(cfc|cfc_i,Cfc_2). 

4  Topic-Dependent  Language  Models 

The  linear  interpolation  of  the  story-dependent  un¬ 
igram  models  (1)  and  (6)  with  a  story-independent 


trigram  model,  as  described  above,  is  very  reminis¬ 
cent  of  monolingual  topic-dependent  language  mod¬ 
els  (cf.  e.g.  (Iyer  and  Ostendorf,  1999)).  This  moti¬ 
vates  us  to  construct  topic-dependent  LMs  and  con¬ 
trast  their  performance  with  these  models. 

To  this  end,  we  represent  each  Chinese  article  in 
the  training  coipus  by  a  bag-of-words  vector,  and 
cluster  the  vectors  using  a  standard  K-means  algo¬ 
rithm.  We  use  random  initialization  to  seed  the  al¬ 
gorithm,  and  a  standard  TF-IDF  weighted  cosine- 
similarity  as  the  “metric”  for  clustering.  We  per¬ 
form  a  few  iterations  of  the  K-means  algorithm,  and 
deem  the  resulting  clusters  as  representing  differ¬ 
ent  topics.  We  then  use  a  bag-of-words  centroid 
created  from  all  the  articles  in  a  cluster  to  repre¬ 
sent  each  topic.  Topic -dependent  trigram  LMs,  de¬ 
noted  Pj(ck  |cfc_i,  Cfc_2),  are  also  computed  for  each 
topic  exclusively  from  the  articles  in  the  j-th  cluster, 
1  <j<  K. 

Each  Mandarin  test  story  is  represented  by  a  bag- 
of-words  vector  P(c\df)  generated  from  the  first- 
pass  ASR  output,  and  the  topic-centroid  ti  having 
the  highest  TF-IDF  weighted  cosine-similarity  to  it 
is  chosen  as  the  topic  of  df .  Topic-dependent  LMs 
are  then  constructed  for  each  story  df  as 

-Plbpic— trigram (c&  |c/c— 1;  Cfc— 2;  ti)  =  (8) 

APtt(cfc|cfc_i,cfc_2)  +  (1  -  A)P(c*|c*_i,c*_2) 

and  used  in  a  second  pass  of  recognition. 

Alternatives  to  topic-dependent  LMs  for  exploit¬ 
ing  long-range  dependencies  include  cache  LMs  and 
monolingual  lexical  triggers;  both  unlikely  to  be  as 
effective  in  the  presence  of  significant  ASR  errors. 

5  ASR  Training  and  Test  Corpora 

We  investigate  the  use  of  the  techniques  described 
above  for  improving  ASR  performance  on  Man¬ 
darin  news  broadcasts  using  English  newswire  texts. 
We  have  chosen  the  experimental  ASR  setup  cre¬ 
ated  in  the  2000  Johns  Hopkins  Summer  Workshop 
to  study  Mandarin  pronunciation  modeling,  exten¬ 
sive  details  about  which  are  available  in  Lung  et 
al  (2000).  The  acoustic  training  data  (~10  hours) 
for  their  ASR  system  was  obtained  from  the  1997 
Mandarin  Broadcast  News  distribution,  and  context- 
dependent  state-clustered  models  were  estimated  us¬ 
ing  initials  and  finals  as  subword  units.  Two  Chinese 


text  corpora  and  an  English  coipus  arc  used  to  esti¬ 
mate  LMs  in  our  experiments.  A  vocabulary  C  of 
51K  Chinese  words,  used  in  the  ASR  system,  is  also 
used  to  segment  the  training  text.  This  vocabulary 
gives  an  OOV  rate  of  5%  on  the  test  data. 

XINHUA:  We  use  the  Xinhua  News  coipus  of 
about  13  million  words  to  represent  the  scenario 
when  the  amount  of  available  LM  training  text  bor¬ 
ders  on  adequate,  and  estimate  a  baseline  trigram 
LM  for  one  set  of  experiments. 

HUB-4NE:  We  also  estimate  a  trigram  model 
from  only  the  96K  words  in  the  transcriptions  used 
for  training  acoustic  models  in  our  ASR  system. 
This  coipus  represents  the  scenario  when  little  or  no 
additional  text  is  available  to  train  LMs. 

NAB-TDT:  English  text  contemporaneous  with 
the  test  data  is  often  easily  available.  Lor  our  test  set, 
described  below,  we  select  (from  the  North  Ameri¬ 
can  News  Text  coipus)  articles  published  in  1997  in 
The  Los  Angeles  Times  and  The  Washington  Post, 
and  articles  from  1998  in  the  New  York  Times  and 
the  Associated  Press  news  service  (from  TDT-2  cor¬ 
pus).  This  amounts  to  a  collection  of  roughly  45,000 
articles  containing  about  30-million  words  of  En¬ 
glish  text;  a  modest  collection  by  CLIR  standards. 

Our  ASR  test  set  is  a  subset  (Lung  et  al  (2000)) 
of  the  NIST  1997  and  1998  HUB-4NE  bench¬ 
mark  tests,  containing  Mandarin  news  broadcasts 
from  three  sources  for  a  total  of  about  9800  words. 
We  generate  two  sets  of  lattices  using  the  baseline 
acoustic  models  and  bigram  LMs  estimated  from 
XINHUA  and  HUB-4NE.  All  our  LMs  are  evaluated 
by  rescoring  300-best  lists  extracted  from  these  two 
sets  of  lattices.  The  300-best  lists  from  the  XINHUA 
bigram  LM  arc  used  in  all  XINHUA  experiments, 
and  those  from  the  HUB-4NE  bigram  LM  in  all 
HUB-4NE  experiments.  We  report  both  word  error 
rates  (WER)  and  character  error  rates  (CER),  the  lat¬ 
ter  being  independent  of  any  difference  in  segmenta¬ 
tion  of  the  ASR  output  and  reference  transcriptions. 

6  ASR  Performance  of  Cross-Lingual  LMs 

We  begin  by  rescoring  the  300-best  lists  from  the 
bigram  lattices  with  trigram  models.  Lor  each  test 
story  dp,  we  perform  CLIR  using  the  first  pass  ASR 
output  to  choose  the  most  similar  English  docu¬ 
ment  df  from  NAB-TDT.  Then  we  create  the  cross- 


lingual  unigram  model  of  (1).  We  also  find  the  inter¬ 
polation  weight  A  which  maximizes  the  likelihood 
of  the  1-best  hypotheses  of  all  test  utterances  from 
the  first  ASR  pass.  Table  1  shows  the  perplexity  and 
WER  for  XINHUA  and  HUB-4NE. 


Language  model 

Peip 

WER 

p-value 

XINHUA  trigram 

426 

49.9% 

- 

CL-interpolated 

375 

49.5% 

0.208 

HUB-4NE  trigram 

1195 

60.1% 

- 

CL-interpolated 

750 

59.3% 

<  0.001 

Table  1:  Word- Perplexity  and  ASR  WER  of  LMs 
based  on  single  English  document  and  global  A. 

All  p- values  reported  in  this  paper  are  based  on 
the  standard  NIST  MAPSSWE  test  (Pallett  et  al., 
1990),  and  indicate  the  statistical  significance  of  a 
WER  improvement  over  the  corresponding  trigram 
baseline,  unless  otherwise  specified. 

Evidently,  the  improvement  brought  by  CL- 
interpolated  LM  is  not  statistically  significant  on 
XINHUA.  On  HUB-4NE  however,  where  Chinese 
text  is  scarce,  the  CL-interpolated  LM  delivers  con¬ 
siderable  benefits  via  the  large  English  coipus. 

6.1  Likelihood-Based  Story-Specific  Selection 
of  Interpolation  Weights  and  the  Number 
of  English  Documents  per  Mandarin  Story 

The  experiments  above  naively  used  the  one  most 
similar  English  document  for  each  Mandarin  story, 
and  a  global  A  in  (2),  no  matter  how  similar  the  best 
matching  English  document  is  to  a  given  Mandarin 
news  story.  Rather  than  choosing  one  most  simi¬ 
lar  English  document  from  NAB-TDT,  it  stands  to 
reason  that  choosing  more  than  one  English  docu¬ 
ment  may  be  helpful  if  many  have  a  high  similarity 
score,  and  perhaps  not  using  even  the  best  matching 
document  may  be  fruitful  if  the  match  is  sufficiently 
poor.  It  may  also  help  to  have  a  greater  interpola¬ 
tion  weight  A  for  stories  with  good  matches,  and  a 
smaller  A  for  others.  For  experiments  in  this  sub¬ 
section,  we  select  a  different  A  for  each  test  story, 
again  based  on  maximizing  the  likelihood  of  the  1- 
best  output  given  a  CL-Unigram  model.  The  other 
issue  then  is  the  choice  and  the  number  of  English 
documents  to  translate. 

N-best  documents:  One  could  choose  a  predeter¬ 
mined  number  N  of  the  best  matching  English  doc¬ 


uments  for  each  Mandarin  story.  We  experimented 
with  values  of  1,  10,  30,  50,  80  and  100,  and  found 
that  N  =  30  gave  us  the  best  LM  performance, 
but  only  marginally  better  than  N  =  1  as  described 
above.  Details  arc  omitted,  as  they  are  uninteresting. 

A11  documents  above  a  similarity  threshold: 
The  argument  against  always  taking  a  predetermined 
number  of  the  best  matching  documents  may  be  that 
it  ignores  the  goodness  of  the  match.  An  alternative 
is  to  take  all  English  documents  whose  similarity  to 
a  Mandarin  story  exceeds  a  certain  predetermined 
threshold.  As  this  threshold  is  lowered,  starting  from 
a  high  value,  the  order  in  which  English  documents 
are  selected  for  a  particular'  Mandarin  story  is  the 
same  as  the  order  when  choosing  the  IV-best  docu¬ 
ments,  but  the  number  of  documents  selected  now 
varies  from  story  to  story.  It  is  possible  that  for 
some  stories,  even  the  best  matching  English  doc¬ 
ument  falls  below  the  threshold  at  which  other  sto¬ 
ries  have  found  more  than  one  good  match.  We  ex¬ 
perimented  with  various  thresholds,  and  found  that 
while  a  threshold  of  0.12  gives  us  the  lowest  per¬ 
plexity  on  the  test  set,  the  reduction  is  insignificant. 
This  points  to  the  need  for  a  story-specific  strategy 
for  choosing  the  number  of  English  documents,  in¬ 
stead  of  a  global  threshold. 

Likelihood-based  selection  of  the  number  of 
English  documents:  Figure  2  shows  the  perplex¬ 
ity  of  the  reference  transcriptions  of  one  typical  test 
story  under  the  LM  (2)  as  a  function  of  the  number 
of  English  documents  chosen  for  creating  (1).  For 
each  choice  of  the  number  of  English  documents, 
the  interpolation  weight  A  in  (2)  is  chosen  to  max¬ 
imize  the  likelihood  (also  shown)  of  the  first  pass 
output.  This  suggests  that  choosing  the  number  of 
English  documents  to  maximize  the  likelihood  of  the 
first  pass  ASR  output  is  a  good  strategy. 

For  each  Mandarin  test  story,  we  choose  the 
1000-best- matching  English  documents  and  divide 
the  dynamic  range  of  their  similarity  scores  evenly 
into  10  intervals.  Next,  we  choose  the  documents 
in  the  top  pj-th  of  the  range  of  similarity  scores, 
not  necessarily  the  top-100  documents,  compute 
-Pcl— unigram(c|df ),  determine  the  A  in  (2)  that  max¬ 
imizes  the  likelihood  of  the  first  pass  output  of  only 
the  utterances  in  that  story,  and  record  this  likeli¬ 
hood.  We  repeat  this  with  documents  in  the  top  pj-th 
of  the  range  of  similarity  scores,  the  top  pj-th,  etc.. 


Figure  2:  Peiplexity  of  the  Reference  Transcription 
and  the  Likelihood  of  the  ASR  Output  v/s  Number 
of  df  for  a  Typical  Test  Story. 


and  obtain  the  likelihood  as  a  function  of  the  simi¬ 
larity  threshold.  We  choose  the  threshold  that  max¬ 
imizes  the  likelihood  of  the  first  pass  output.  Thus 
the  number  of  English  documents  df  in  (1),  as  well 
as  the  interpolation  weight  A  in  (2),  arc  chosen  dy¬ 
namically  for  each  Mandarin  story  to  maximize  the 
likelihood  of  the  ASR  output.  Table  2  shows  ASR 
results  for  this  likelihood-based  story-specific  adap¬ 
tation  scheme. 

Note  that  significant  WER  improvements  arc 
obtained  from  the  CL-interpolated  LM  using 
likelihood-based  story-specific  adaptation  even  for 
the  case  of  the  XINHUA  LM.  Furthermore,  the  per¬ 
formance  of  the  CL-interpolated  LM  is  even  better 
than  the  topic -dependent  LM.  This  is  remarkable, 
since  the  CL-interpolated  LM  is  based  on  unigram 
statistics  from  English  documents,  while  the  topic- 
trigram  LM  is  based  on  trigram  statistics.  We  be¬ 
lieve  that  the  contemporaneous  and  story-specific 
nature  of  the  English  document  leads  to  its  rela¬ 
tively  higher  effectiveness.  Our  conjecture,  that  the 
contemporaneous  cross-lingual  statistics  and  static 
topic-trigram  statistics  are  complementary,  is  sup¬ 
ported  by  the  significant  further  improvement  in 
WER  obtained  by  the  interpolation  of  the  two  LMs, 
as  shown  on  the  last  line  for  XINHUA. 

The  significant  gain  in  ASR  performance  in  the 
resource  deficient  HUB-4NE  case  arc  obvious.  The 
small  size  of  the  HUB-4NE  coipus  makes  topic- 
models  ineffective. 


6.2  Comparison  of  Cross-Lingual  Triggers 
with  Stochastic  Translation  Dictionaries 

Once  we  select  cross-lingual  trigger-pairs  as  de¬ 
scribed  in  Section  3,  Pr(c\e)  in  (1)  is  replaced  by 
-Prrig(c|e)  of  (5),  and  Pr(e\c )  in  (3)  by  PTrig(e|c). 
Therefore,  given  a  set  of  cross-lingual  trigger-pairs, 
the  trigger-based  models  are  free  from  requiring 
a  translation  lexicon.  Furthermore,  a  document- 
aligned  comparable  coipus  is  all  that  is  required  to 
construct  the  set  of  trigger-pairs.  We  otherwise  fol¬ 
low  the  same  experimental  procedure  as  above. 

As  Table  2  shows,  the  trigger-based  model  (Trig- 
interpolated)  performs  only  slightly  worse  than  the 
CL-interpolated  model.  One  explanation  for  this 
degradation  is  that  the  CL-interpolated  model  is 
trained  from  the  sentence-aligned  coipus  while  the 
trigger-based  model  is  from  the  document-aligned 
coipus.  There  are  two  steps  which  could  be  affected 
by  this  difference,  one  being  CLIR  and  the  other  be¬ 
ing  the  translation  of  the  df 's  into  Chinese.  Some 
errors  in  CLIR  may  however  be  masked  by  our 
likelihood-based  story-specific  adaptation  scheme, 
since  it  finds  optimal  retrieval  settings,  dynamically 
adjusting  the  number  of  English  documents  as  well 
as  the  interpolation  weight,  even  if  CLIR  performs 
somewhat  suboptimally.  Furthermore,  a  document- 
aligned  coipus  is  much  easier  to  build.  Thus  a  much 
bigger  and  more  reliable  comparable  coipus  may  be 
used,  and  eventually  more  accurate  trigger-pairs  will 
be  acquired. 

We  note  with  some  satisfaction  that  even  simple 
trigger-pairs  selected  on  the  basis  of  mutual  infor¬ 
mation  are  able  to  achieve  peiplexity  and  WER  re¬ 
ductions  comparable  to  a  stochastic  translation  lex¬ 
icon:  the  smallest  p-  value  at  which  the  difference 
between  the  WERs  of  the  CL-interpolated  LM  and 
the  Trig-interpolated  LM  in  Table  2  would  be  signif¬ 
icant  is  0.4  for  XINHUA  and  0.7  for  HUB-4NE. 

Triggers  (4)  vs  (5):  We  compare  the  alternative 
Pxrig  (•  |  •)  definitions  (4)  and  (5)  for  replacing  Pt(:\) 
in  (1).  The  resulting  CL-interpolated  LM  (2)  yields  a 
peiplexity  of  370  on  the  XINHUA  test  set  using  (4), 
compared  to  367  using  (5).  Similarly,  on  the  HUB- 
4NE  test  set,  using  (4)  yields  736,  while  (5)  yields 
727.  Therefore,  (5)  has  been  used  throughout. 


XINHUA  HUB-4NE 


Peip 

WER 

CER 

p-value 

Language  model 

Peip 

WER 

CER 

p- value 

426 

49.9% 

28.8% 

- 

Baseline  Trigram 

1195 

60.1% 

44.1% 

- 

381 

49.1% 

28.4% 

0.003 

Topic-trigram 

1122 

60.0% 

44.1% 

0.660 

367 

49.1% 

28.6% 

0.004 

Trig-interpolated 

727 

58.8% 

43.3% 

<  0.001 

346 

48.8% 

28.4% 

<  0.001 

CL-interpolated 

630 

58.8% 

43.1% 

<  0.001 

340 

48.7% 

28.4% 

<  0.001 

Topic  +  Trig-interpolated 

730 

59.2% 

43.5% 

0.002 

326 

48.5% 

28.2% 

<  0.001 

Topic  +  CL-interpolated 

631 

59.0% 

43.3% 

<  0.001 

320 

48.3% 

28.1% 

<  0.001 

Topic  +  Trig-  +  CL-inteip. 

627 

59.0% 

43.3% 

<  0.001 

Table  2:  Peiplexity  and  ASR  Performance  with  a  Likelihood-Based  Story- Specific  Selection  of  the  Number 
of  English  Documents  df’ s  and  Interpolation  Weight  A  for  Each  Mandarin  Story. 


7  Conclusions  and  Future  Work 

We  have  demonstrated  a  statistically  significant  im¬ 
provement  in  ASR  WER  (1.4%  absolute)  and  in 
peiplexity  (23%)  by  exploiting  cross-lingual  side- 
information  even  when  nontrivial  amount  of  train¬ 
ing  data  is  available,  as  seen  on  the  XINHUA  cor¬ 
pus.  Our  methods  are  even  more  effective  when  LM 
training  text  is  hard  to  come  by  in  the  language  of 
interest:  47%  reduction  in  peiplexity  and  1.3%  ab¬ 
solute  in  WER  as  seen  on  the  HUB-4NE  coipus. 
Most  of  these  gains  come  from  the  optimal  choice  of 
adaptation  parameters.  The  ASR  test  data  we  used 
in  our  experiments  is  derived  from  a  different  source 
than  the  coipus  on  which  the  translation  and  trigger 
models  are  trained,  and  the  techniques  work  even 
when  the  bilingual  coipus  is  only  document-aligned, 
which  is  a  realistic  reflection  of  the  situation  in  a 
resource-deficient  language. 

We  arc  developing  maximum  entropy  models  to 
more  effectively  combine  the  multiple  information 
sources  we  have  used  in  our  experiments,  and  expect 
to  report  the  results  in  the  near  future. 
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